Search indexing using discourse trees

ABSTRACT

Systems, devices, and methods of the present invention create a searchable index that includes informative portions of text. In an example, a computer-implemented method creates a discourse tree from a body of text. For each non-terminal node in the discourse tree, the method identifies a rhetorical relationship associated with the non-terminal node. The method labels each terminal node associated with the non-terminal node as either a nucleus or a satellite. The method further accesses a rule associated with the rhetorical relationship, and selects, based on the rule, selects the fragment associated with the nucleus. The method creates a searchable index including the selected fragments.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/142,759 filed Sep. 26, 2018, the contents of which is herebyincorporated by reference in its entirety, which claims the benefit ofU.S. Provisional Application No. 62/563,806, filed Sep. 27, 2017, whichis incorporated by reference in its entirety.

TECHNICAL FIELD

This disclosure is generally concerned with linguistics. Morespecifically, this disclosure relates to using communicative discoursetrees to improve search indexing.

BACKGROUND

Much online content is available via question-answer pairs such asfrequently-asked questions stored on customer portals or internalcompany portals. Question-answer pairs can be an efficient manner tofamiliarize a user with content. In some cases, autonomous agents(ChatBots) can import such question-answer pairs in order to field userquestions.

But such question-answer pairs can contain content that is not centralto a topic of an answer. For example, content can include text that isirrelevant or misleading, non-responsive to the particular question, oris neutral and not helpful. If irrelevant text is indexed by akeyword-based search engine, the precision of the search engine islowered. Moreover, an autonomous agent attempting to answer a userquestion based on erroneously-indexed text may answer the questionincorrectly, resulting in lowered user confidence in the agent.

As such, solutions are needed for identifying informative text from textto be indexed.

BRIEF SUMMARY

Generally, systems, devices, and methods of the present invention usediscourse trees to determine informative parts of an answer. In anexample, a method accesses a body of text including fragments. Themethod creates a discourse tree from the body of text. The discoursetree includes nodes. Each nonterminal node represents a rhetoricalrelationship between two of the fragments. Each terminal node isassociated with one or more fragments and is associated with anon-terminal node. The method identifies a rhetorical relationshipassociated with the non-terminal node for each non-terminal node. Themethod labels each terminal node associated with the non-terminal nodeas either a nucleus or a satellite, accesses a rule associated with therhetorical relationship. The rule specifies accessing a fragment of textassociated with one or more of the nucleus or the satellite. The methodselects, based on the rule, one or more of the fragment associated withthe nucleus or the fragment associated with the satellite. The methodcreates a searchable index including multiple entries, each entrycorresponding to a selected fragment.

In an aspect, each entry can represent a question that can be answeredby the body of text or a phrase to be matched to the question.

In an aspect, the rule specifies selecting the fragments associated witha nucleus and discarding fragments associated with a satellite.

In a further aspect, one or more fragments comprise clauses of compoundsentences.

In yet another aspect, each rhetorical relationship is one of anelaboration, enablement, condition, contrast, or attribution.

In yet another aspect, the method, for at least one non-terminal node,identifies the rhetorical relationship as either a same unit or a jointrelationship and selects both the nucleus and the satellite. The rulespecifies accessing both the nucleus and the satellite.

In yet another aspect, the method, for at least one non-terminal node,identifies the rhetorical relationship as an attribution relationship,and responsive to determining that the rhetorical relationship does notrepresent a query by an author, selects the nucleus.

In yet another aspect, the method creates an index of additionalquestions by determining, for each entry of the searchable index, anadditional question that is answered by the entry. The method receives aquery from an external device. The method searches in each of thequestions of multiple question-answer pairs, for matches correspondingto the query. The method, responsive to determining that fewer than athreshold number of matches are available, searches, in the index ofadditional questions, for additional matches to the query. The method,responsive to determining that fewer than a threshold number ofadditional matches are available in the index of additional questions,searches for further matches in each answers of the plurality ofquestion-answer pairs. The method provides a response to the externaldevice.

In yet another aspect, the method accesses question-answer pairs, eachanswer includes fragments. The method creates, for each answer, adiscourse tree. The discourse tree includes nodes, each nonterminal noderepresenting a rhetorical relationship between two of the fragments andeach terminal node is associated with one or more fragments. For eachnon-terminal node in each answer, the method identifies a rhetoricalrelationship associated with the non-terminal node. The method labelseach terminal node associated with the non-terminal node as either anucleus or a satellite. The method accesses a rule associated with therhetorical relationship. The rule specifies accessing a fragment of textassociated with either the nucleus or the satellite. The method selects,based on the rule, one or more of the fragment associated with thenucleus or the fragment associated with the satellite. The methodcreates a searchable index of additional questions, the searchable indexincluding multiple entries, each entry corresponding to one of theselected fragments for the answers.

In yet another aspect, the method receives a query from an externaldevice. The method searches in each of the questions of multiplequestion-answer pairs, for matches corresponding to the query. Themethod, responsive to determining that fewer than a threshold number ofmatches are available, searches in the searchable index of additionalquestions, for additional matches to the query. The method, responsiveto determining that fewer than a threshold number of additional matchesare available in the searchable index of additional questions, searchesfor further matches in each answer of the plurality of question-answerpairs. The method provides a response to the external device.

In yet another aspect, a method trains a classification model todetermine text to index. The method accesses a set of training dataincluding a set of training pairs. Each training data pair includes textand an expected classification. The set of training data includes both afirst training data pair that has an expected classification of anucleus and a second training data pair that has an expectedclassification of a satellite. The method trains a classification modelby iteratively providing one of the training data pairs to theclassification model, receiving, from the classification model, adetermined classification, calculating a loss function by calculating adifference between the determined classification and the expectedclassification, adjusting internal parameters of the classificationmodel to minimize the loss function.

The above methods can be implemented on tangible computer-readable mediaand/or operating within a computer processor and attached memory.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an exemplary rhetoric classification environment inaccordance with an aspect.

FIG. 2 depicts an example of a discourse tree in accordance with anaspect.

FIG. 3 depicts a further example of a discourse tree in accordance withan aspect.

FIG. 4 depicts illustrative schemas in accordance with an aspect.

FIG. 5 depicts a node-link representation of the hierarchical binarytree in accordance with an aspect.

FIG. 6 depicts an exemplary indented text encoding of the representationin FIG. 5 in accordance with an aspect.

FIG. 7 depicts an exemplary discourse tree for an example request aboutproperty tax in accordance with an aspect.

FIG. 8 depicts an exemplary response for the question represented inFIG. 7 .

FIG. 9 illustrates a discourse tree for an official answer in accordancewith an aspect.

FIG. 10 illustrates a discourse tree for a raw answer in accordance withan aspect.

FIG. 11 illustrates a discourse tree for an example of an answer to aquestion, in accordance with an aspect.

FIG. 12 depicts a flowchart illustrating an example of a process fordetermining informative text for indexing, in accordance with an aspect.

FIG. 13 depicts a flowchart illustrating an example of a process forresponding to searches, in accordance with an aspect.

FIG. 14 depicts a parse tree for a sentence used to illustrate forming aquestion from text, in accordance with an aspect.

FIG. 15 illustrates an autonomous agent answering user questions inaccordance with an aspect.

FIG. 16 depicts a flowchart illustrating an example of a process fortraining a classification model to determine informative parts of textfor indexing, in accordance with an aspect.

FIG. 17 depicts a simplified diagram of a distributed system forimplementing one of the aspects.

FIG. 18 is a simplified block diagram of components of a systemenvironment by which services provided by the components of an aspectsystem may be offered as cloud services in accordance with an aspect.

FIG. 19 illustrates an exemplary computer system, in which variousaspects of the present invention may be implemented.

DETAILED DESCRIPTION

As discussed above, existing search solutions index entire bodies oftext without regard to which parts of the text are informative, therebycausing erroneous results and lower search precision. In contrast,aspects described herein use discourse trees to identify informativefragments of text for indexing. In so doing, certain aspects improve theprecision of search engines and autonomous agents that answer questionsfrom users. Precision refers to a fraction of informative instancesamong retrieved instances.

Other aspects use trained machine learning models to identifyinformative fragments of text for indexing, permitting the applicationof sets of rules that dictate which parts of a discourse tree areinformative for indexing. Further aspects generate alternative questionsfrom the indexed text that can be answered by the body of text, therebyimproving search recall (the fraction of informative instances that havebeen retrieved of the total amount of informative instances).

Discourse trees originate from Rhetorical Structure Theory (RST). RSTmodels a logical organization of text employed by an author, relying onrelations between parts of text. RST simulates text coherence by forminga hierarchical, connected structure of texts via discourse trees.Rhetoric relations are split into the classes of coordinate andsubordinate; these relations hold across two or more text spans andtherefore implement coherence. These text spans are called elementarydiscourse units (EDUs). Clauses in a sentence and sentences in a textare logically connected by the author. The meaning of a given sentenceis related to that of the previous and the following sentences.

The leaves of a discourse tree correspond to EDUs, the contiguous atomictext spans. Adjacent EDUs are connected by coherence relations (e.g.,attribution, sequence), forming higher-level discourse units. The leavesof a particular EDU are logically related. This relationship is referredto as the coherence structure of the text. Example relations includeelaboration and enablement. As used herein, “nuclearity” refers to whichtext segment, fragment, or span, is more central to an author's purpose.A “nucleus” refers to a span of text that is more central to an author'spurpose than a “satellite,” which is less central to an author'spurpose. Certain aspects use the determined EDUs of a discourse tree forto a body of text and the relations between the EDUs to determine whichEDUs should be indexed for search. Different relations (e.g.,elaboration, contrast, etc.) can employ different rules.

The following non-limiting example is provided to introduce certainaspects. An indexing application receives the following text that is areview of a digital camera: “it shoots well in low light, and I made afew good shots on a boat at night.” This text would be a good answer forthe following questions: “Which camera suits well in low light?,” “Howto shoot in low light,” “low light camera,” and “low light conditions.”In contrast, this text is not a good answer for search queries such as“good shorts at a boat,” “night boat,” “boat at night,” “good shots,”“good boat,” or “good night.”

Creating a discourse tree for the above compound sentence revealsnucleus “it shoots well in low light,” and satellite “I made a few goodshots on a boat at night.” The nucleus and satellite are related by an“elaboration” relation because the satellite fragment elaborates thenucleus statement. As can be seen with the “elaboration” relationship,the nucleus fragment provides useful information while the satellitefragment does not. In particular, the nucleus phrase is more importanttext is strongly correlated with the potential question. In contrast,the satellite phrase provides less important details which are lesslikely to be asked directly. If this less important text is indexed, itmight trigger this answer as a response to question on something totallydifferent. Certain aspects use trained machine-learning models topredict which fragments within a body of text should be indexed forsearch based in part on the type of rhetorical relationship associatedwith the particular fragment. These aspects train machine learningmodels to predict informative EDUs to be indexed.

Technical advantages the present disclosure include improved searchengine performance (precision, recall) and improved autonomous agents ascompared to traditional statistical-based approaches. As discussed,existing keyword-based solutions erroneously index text that is not onlynon-informative but irrelevant. Considering the above example, existingsearch techniques may identify words and phrases from the original textsuch as “boat” or “good shots” as significant, thereby causing a searchengine to erroneously index and rely on these phrases.

Certain Definitions

As used herein, “rhetorical structure theory” is an area of research andstudy that provided a theoretical basis upon which the coherence of adiscourse could be analyzed.

As used herein, “discourse tree” or “DT” refers to a structure thatrepresents the rhetorical relations for a sentence of part of asentence.

As used herein, a “rhetorical relation,” “rhetorical relationship,” or“coherence relation” or “discourse relation” refers to how two segmentsof discourse are logically connected to one another. Examples ofrhetorical relations include elaboration, contrast, and attribution.

As used herein, a “sentence fragment,” or “fragment” is a part of asentence that can be divided from the rest of the sentence. A fragmentis an elementary discourse unit. For example, for the sentence “Dutchaccident investigators say that evidence points to pro-Russian rebels asbeing responsible for shooting down the plane,” two fragments are “Dutchaccident investigators say that evidence points to pro-Russian rebels”and “as being responsible for shooting down the plane.” A fragment can,but need not, include a verb.

As used herein, “index” is a table, data structure, pointer, or othermechanism that links two keywords, data, or parts of text for purposesof data retrieval. For example, an index can include links to keywords,data or parts of text. Index can be a string match or an inverse index.

Turning now to the figures, FIG. 1 shows an exemplary indexingenvironment in accordance with an aspect. FIG. 1 depicts computingdevice 101, input text 130, user question 140, answer 150, data network104, server 160, and mobile device 170. Computing device 101 includesone or more of indexing application 102, question-answer index 135,machine-learning model 120, and training data 125. Computing device 101can receive text and use discourse trees and/or machine learning modelsto determine informative parts of text to index, and can use an index tomore accurately answer questions received via an autonomous agent.

More specifically, indexing application 102 can determine informativefragments of input text 130, index the determined informative fragments,generate additional questions, generate a question-answer index 135, orgenerate answer 150 using question-answer index 135.

In an example, indexing application 102 receives a body of input text130 that is directed to a particular news story with comments. Inputtext 130 contains multiple sentences that broadly speaking, address thetopic of Hurricane “Lane” that made land on Hawaii, but the commentsinclude irrelevant, opinionated, or incorrect statements such as“residents of Hawaii deserve this, who lives on an island?” Indexingapplication 102 creates a discourse tree from input text 130. Thediscourse tree includes identification of different fragments of text asnuclei or satellites, some of which are as shown in Table 1 below.

TABLE 1 Nucleus Satellite Officials also expressed concern which theysaid have already about the spread of brush fires. scorched more than2,000 acres and destroyed several homes. Dozens of residents have beenwading through waist-high water, forced to evacuate their homes andsometimes spreading brush over the past two days. fires, to safety,while thousands of others remain in shelters Federal EmergencyManagement mostly in Oahu officials said about 2,000 people remain inshelters.

Indexing application 102 applies a set of rules that govern whether, fora specific rhetoric relation, a nucleus or a satellite EDUs should beindexed. Indexing application 102 extracts the EDUs (fragments of text)that are informative. Indexing application 102 creates a question foreach indexed fragment of text that can be answered by the particularfragment of text. Indexing application creates question-answer index135, which includes multiple entries. Each entry includes an indexedfragment of text and the corresponding generated question.

Subsequently, indexing application 102 receives a question from a uservia an autonomous agent (ChatBot). The question states “how many peopleremain in shelters?” Indexing application 102 accesses thequestion-answer index 135, identifying the corresponding EDU “FederalEmergency Management officials said about 2,000 people remain inshelters.” Indexing application 102 provides this answer to the userdevice.

Mobile device 170 can be any mobile device such as a mobile phone, smartphone, tablet, laptop, smart watch, and the like. Mobile device 170communicates via data network 104 to server 160 or computing device 101.In this manner, mobile device 170 can provide questions from and outputanswers to a user, to server 160 or computing device 101. Data network104 can be any public or private network, wired or wireless network,Wide Area Network, Local Area Network, or the Internet.

In a further aspect, indexing application 102 uses a trainedmachine-learning model 120 to predict informative parts of fragments toindex. An example of machine-learning model 120 is a classificationmodel. Machine-learning model 120 is trained with training data 125,which can include sets of positive-negative training pairs. Positivetraining pairs include text or text fragments identified as informativefor indexing and negative training pairs include text or text fragmentsidentified as not informative.

Rhetoric Structure Theory and Discourse Trees

Linguistics is the scientific study of language. For example,linguistics can include the structure of a sentence (syntax), e.g.,subject-verb-object, the meaning of a sentence (semantics), e.g. dogbites man vs. man bites dog, and what speakers do in conversation, i.e.,discourse analysis or the analysis of language beyond the sentence.

The theoretical underpinnings of discourse, Rhetoric Structure Theory(RST), can be attributed to Mann, William and Thompson, Sandra,“Rhetorical structure theory: A Theory of Text organization,”Text-Interdisciplinary Journal for the Study of Discourse, 8(3):243-281,1988. Similar to how the syntax and semantics of programming languagetheory helped enable modern software compilers, RST helped enabled theanalysis of discourse. More specifically RST posits structural blocks onat least two levels, a first level such as nuclearity and rhetoricalrelations, and a second level of structures or schemas. Discourseparsers or other computer software can parse text into a discourse tree.

Rhetoric Relations

As discussed, aspects described herein use rhetorical relations anddiscourse trees. Rhetorical relations can be described in differentways. For example, Mann and Thompson describe twenty-three possiblerelations. C. Mann, William & Thompson, Sandra. (1987) (“Mann andThompson”). Rhetorical Structure Theory: A Theory of Text Organization.Other numbers of relations are possible. Table 2 below lists differentrhetorical relations.

TABLE 2 Relation Name Nucleus Satellite Antithesis ideas favored by theideas disfavored by author the author Background text whose text forfacilitating understanding is being understanding facilitatedCircumstance text expressing the an interpretive context of events orideas situation or time occurring in the interpretive context Concessionsituation affirmed by situation which is author apparently inconsistentbut also affirmed by author Condition action or situation conditioningsituation whose occurrence results from the occurrence of theconditioning situation Elaboration basic information additionalinformation Enablement an action information intended to aid the readerin performing an action Evaluation a situation an evaluative commentabout the situation Evidence a claim information intended to increasethe reader's belief in the claim Interpretation a situation aninterpretation of the situation Justify text information supporting thewriter's right to express the text Motivation an action informationintended to increase the reader's desire to perform the action Non- asituation another situation which causes volitional that one, but not byanyone's Cause deliberate action Non- a situation another situationwhich is volitional caused by that one, but not by Result anyone'sdeliberate action Otherwise action or situation conditioning situation(anti whose occurrence conditional) results from the lack of occurrenceof the conditioning situation Purpose an intended situation the intentbehind the situation Restatement a situation a reexpression of thesituation Solutionhood a situation or method a question, request,problem, supporting full or partial or other expressed need satisfactionof the need Summary text a short summary of that text Volitional asituation another situation which causes Cause that one, by someone'sdeliberate action Volitional a situation another situation which iscaused Result by that one, by someone's deliberate action

Some empirical studies postulate that the majority of text is structuredusing nucleus-satellite relations. See Mann and Thompson. But otherrelations do not carry a definite selection of a nucleus. Examples ofsuch relations are shown in Table 3 below.

TABLE 3 Relation Name Span Other Span Contrast One alternate The otheralternate Joint (unconstrained) (unconstrained) List An item A next itemSequence An item A next item

FIG. 2 depicts an example of a discourse tree in accordance with anaspect. FIG. 2 includes discourse tree 200. Discourse tree includes textspan 201, text span 202, text span 203, relation 210 and relation 211.The numbers in FIG. 2 correspond to the three text spans. FIG. 3corresponds to the following example text with three text spans numbered1, 2, 3:

1. Honolulu, Hi. will be site of the 2017 Conference on Hawaiian History

2. It is expected that 200 historians from the U.S. and Asia will attend

3. The conference will be concerned with how the Polynesians sailed toHawaii

For example, relation 210, or elaboration, describes the relationshipbetween text span 201 and text span 202. Relation 210 depicts therelationship, elaboration, between text span 203 and 204. As depicted,text spans 202 and 203 elaborate further on text span 201. In the aboveexample, given a goal of notifying readers of a conference, text span 1is the nucleus. Text spans 2 and 3 provide more detail about theconference. In FIG. 2 , a horizontal number, e.g., 1-3, 1, 2, 3 covers aspan of text (possibly made up of further spans); a vertical linesignals the nucleus or nuclei; and a curve represents a rhetoricrelation (elaboration) and the direction of the arrow points from thesatellite to the nucleus. If the text span only functions as a satelliteand not as a nuclei, then deleting the satellite would still leave acoherent text. If from FIG. 2 one deletes the nucleus, then text spans 2and 3 are difficult to understand.

FIG. 3 depicts a further example of a discourse tree in accordance withan aspect. FIG. 3 includes components 301 and 302, text spans 305-307,relation 310 and relation 311. Relation 310 depicts relation 310,enablement, between components 306 and 305, and 307, and 305. FIG. 3refers to the following text spans:

1. The new Tech Report abstracts are now in the journal area of thelibrary near the abridged dictionary.

2. Please sign your name by any means that you would be interested inseeing.

3. Last day for sign-ups is 31 May.

As can be seen, relation 310 depicts the relationship between entity 307and 306, which is enablement. FIG. 3 illustrates that while nuclei canbe nested, there exists only one most nuclear text span.

Constructing a Discourse Tree

Discourse trees can be generated using different methods. A simpleexample of a method to construct a DT bottom up is:

(1) Divide the discourse text into units by:

-   -   (a) Unit size may vary, depending on the goals of the analysis    -   (b) Typically, units are clauses

(2) Examine each unit, and its neighbors. Is there a relation holdingbetween them?

(3) If yes, then mark that relation.

(4) If not, the unit might be at the boundary of a higher-levelrelation. Look at relations holding between larger units (spans).

(5) Continue until all the units in the text are accounted for.

Mann and Thompson also describe the second level of building blockstructures called schemas applications. In RST, rhetoric relations arenot mapped directly onto texts; they are fitted onto structures calledschema applications, and these in turn are fitted to text. Schemaapplications are derived from simpler structures called schemas (asshown by FIG. 4 ). Each schema indicates how a particular unit of textis decomposed into other smaller text units. A rhetorical structure treeor DT is a hierarchical system of schema applications. A schemaapplication links a number of consecutive text spans, and creates acomplex text span, which can in turn be linked by a higher-level schemaapplication. RST asserts that the structure of every coherent discoursecan be described by a single rhetorical structure tree, whose top schemacreates a span encompassing the whole discourse.

FIG. 4 depicts illustrative schemas in accordance with an aspect. FIG. 4shows a joint schema is a list of items consisting of nuclei with nosatellites. FIG. 4 depicts schemas 401-406. Schema 401 depicts acircumstance relation between text spans 410 and 411. Scheme 402 depictsa sequence relation between text spans 420 and 421 and a sequencerelation between text spans 421 and 422. Schema 403 depicts a contrastrelation between text spans 430 and 431. Schema 404 depicts a jointrelationship between text spans 440 and 441. Schema 405 depicts amotivation relationship between 450 and 451, and an enablementrelationship between 452 and 451. Schema 406 depicts joint relationshipbetween text spans 460 and 462. An example of a joint scheme is shown inFIG. 4 for the three text spans below:

1. Skies will be partly sunny in the New York metropolitan area today.

2. It will be more humid, with temperatures in the middle 80's.

3. Tonight will be mostly cloudy, with the low temperature between 65and 70.

While FIGS. 2-4 depict some graphical representations of a discoursetree, other representations are possible.

FIG. 5 depicts a node-link representation of the hierarchical binarytree in accordance with an aspect. As can be seen from FIG. 5 , theleaves of a DT correspond to contiguous non-overlapping text spanscalled Elementary Discourse Units (EDUs). Adjacent EDUs are connected byrelations (e.g., elaboration, attribution . . . ) and form largerdiscourse units, which are also connected by relations. “Discourseanalysis in RST involves two sub-tasks: discourse segmentation is thetask of identifying the EDUs, and discourse parsing is the task oflinking the discourse units into a labeled tree.” See Joty, Shafiq R andGiuseppe Carenini, Raymond T Ng, and Yashar Mehdad. 2013. Combiningintra- and multi-sentential rhetorical parsing for document-leveldiscourse analysis. In ACL (1), pages 486-496.

FIG. 5 depicts text spans that are leaves, or terminal nodes, on thetree, each numbered in the order they appear in the full text, shown inFIG. 6 . FIG. 5 includes tree 500. Tree 500 includes, for example, nodes501-507. The nodes indicate relationships. Nodes are non-terminal, suchas node 501, or terminal, such as nodes 502-507. As can be seen, nodes503 and 504 are related by a joint relationship. Nodes 502, 505, 506,and 508 are nuclei. The dotted lines indicate that the branch or textspan is a satellite. The relations are nodes in gray boxes.

FIG. 6 depicts an exemplary indented text encoding of the representationin FIG. 5 in accordance with an aspect. FIG. 6 includes text 600 andtext sequences 602-604. Text 600 is presented in a manner more amenableto computer programming. Text sequence 602 corresponds to node 502,sequence 603 corresponds to node 503, and sequence 604 corresponds tonode 504. In FIG. 6 , “N” indicates a nucleus and “S” indicates asatellite.

Examples of Discourse Parsers

Automatic discourse segmentation can be performed with differentmethods. For example, given a sentence, a segmentation model identifiesthe boundaries of the composite elementary discourse units by predictingwhether a boundary should be inserted before each particular token inthe sentence. For example, one framework considers each token in thesentence sequentially and independently. In this framework, thesegmentation model scans the sentence token by token, and uses a binaryclassifier, such as a support vector machine or logistic regression, topredict whether it is appropriate to insert a boundary before the tokenbeing examined. In another example, the task is a sequential labelingproblem. Once text is segmented into elementary discourse units,sentence-level discourse parsing can be performed to construct thediscourse tree. Machine learning techniques can be used.

In one aspect of the present invention, two Rhetorical Structure Theory(RST) discourse parsers are used: CoreNLPProcessor which relies onconstituent syntax, and FastNLPProcessor which uses dependency syntax.See Surdeanu, Mihai & Hicks, Thomas & Antonio Valenzuela-Escarcega,Marco. Two Practical Rhetorical Structure Theory Parsers. (2015).

In addition, the above two discourse parsers, i.e., CoreNLPProcessor andFastNLPProcessor use Natural Language Processing (NLP) for syntacticparsing. For example, the Stanford CoreNLP gives the base forms ofwords, their parts of speech, whether they are names of companies,people, etc., normalize dates, times, and numeric quantities, mark upthe structure of sentences in terms of phrases and syntacticdependencies, indicate which noun phrases refer to the same entities.Practically, RST is a still theory that may work in many cases ofdiscourse, but in some cases, it may not work. There are many variablesincluding, but not limited to, what EDU's are in a coherent text, i.e.,what discourse segmenters are used, what relations inventory is used andwhat relations are selected for the EDUs, the corpus of documents usedfor training and testing, and even what parsers are used. So forexample, in Surdeanu, et al., “Two Practical Rhetorical Structure TheoryParsers,” paper cited above, tests must be run on a particular corpususing specialized metrics to determine which parser gives betterperformance. Thus unlike computer language parsers which givepredictable results, discourse parsers (and segmenters) can giveunpredictable results depending on the training and/or test text corpus.Thus, discourse trees are a mixture of the predicable arts (e.g.,compilers) and the unpredictable arts (e.g., like chemistry whereexperimentation is needed to determine what combinations will give youthe desired results).

In order to objectively determine how good a Discourse analysis is, aseries of metrics are being used, e.g., Precision/Recall/F1 metrics fromDaniel Marcu, “The Theory and Practice of Discourse Parsing andSummarization,” MIT Press, (2000). Precision, or positive predictivevalue is the fraction of informative instances among the retrievedinstances, while recall (also known as sensitivity) is the fraction ofinformative instances that have been retrieved over the total amount ofinformative instances. Both precision and recall are therefore based onan understanding and measure of relevance. Suppose a computer programfor recognizing dogs in photographs identifies eight dogs in a picturecontaining 12 dogs and some cats. Of the eight dogs identified, fiveactually are dogs (true positives), while the rest are cats (falsepositives). The program's precision is 5/8 while its recall is 5/12.When a search engine returns 30 pages only 20 of which were informativewhile failing to return 40 additional informative pages, its precisionis 20/30=2/3 while its recall is 20/60=1/3. Therefore, in this case,precision is ‘how useful the search results are’, and recall is ‘howcomplete the results are.’ The F1 score (also F-score or F-measure) is ameasure of a test's accuracy. It considers both the precision and therecall of the test to compute the score:F1=2×((precision×recall)/(precision+recall)) and is the harmonic mean ofprecision and recall. The F1 score reaches its best value at 1 (perfectprecision and recall) and worst at 0.

Analyzing Request and Response Pairs

FIG. 7 depicts an exemplary discourse tree for an example request aboutproperty tax in accordance with an aspect. The node labels are therelations and the arrowed line points to the satellite. The nucleus is asolid line. FIG. 7 depicts the following text.

Request: “My husbands' grandmother gave him his grandfather's truck. Shesigned the title over but due to my husband having unpaid fines on hislicense, he was not able to get the truck put in his name. I wanted toput in my name and paid the property tax and got insurance for thetruck. By the time it came to sending off the title and getting the tag,I didn't have the money to do so. Now, due to circumstances, I am notgoing to be able to afford the truck. I went to the insurance place andwas refused a refund. I am just wondering that since I am not going tohave a tag on this truck, is it possible to get the property taxrefunded?”

Response: “The property tax is assessed on property that you own. Justbecause you chose to not register it does not mean that you don't ownit, so the tax is not refundable. Even if you have not titled thevehicle yet, you still own it within the boundaries of the tax district,so the tax is payable. Note that all states give you a limited amount oftime to transfer title and pay the use tax. If you apply late, therewill be penalties on top of the normal taxes and fees. You don't need toregister it at the same time, but you absolutely need to title it withinthe period of time stipulated in state law.”

As can be seen in FIG. 7 , analyzing the above text results in thefollowing. “My husbands' grandmother gave him his grandfather's truck”is elaborated by “She signed the title over but due to my husband”elaborated by “having unpaid fines on his license, he was not able toget the truck put in his name.” which is elaborated by “I wanted to putin my name,” “and paid the property tax”, and “and got insurance for thetruck.”

“My husbands' grandmother gave him his grandfather's truck. She signedthe title over but due to my husband having unpaid fines on his license,he was not able to get the truck put in his name. I wanted to put in myname and paid the property tax and got insurance for the truck.” iselaborated by;

“I didn't have the money” elaborated by “to do so” contrasted with

“By the time” elaborated by “it came to sending off the title”

“and getting the tag”

“My husbands' grandmother gave him his grandfather's truck. She signedthe title over but due to my husband having unpaid fines on his license,he was not able to get the truck put in his name. I wanted to put in myname and paid the property tax and got insurance for the truck. By thetime it came to sending off the title and getting the tag, I didn't havethe money to do so” is contrasted with

“Now, due to circumstances,” elaborated with “I am not going to be ableto afford the truck.” which is elaborated with

“I went to the insurance place”

“and was refused a refund”

“My husbands' grandmother gave him his grandfather's truck. She signedthe title over but due to my husband having unpaid fines on his license,he was not able to get the truck put in his name. I wanted to put in myname and paid the property tax and got insurance for the truck. By thetime it came to sending off the title and getting the tag, I didn't havethe money to do so. Now, due to circumstances, I am not going to be ableto afford the truck. I went to the insurance place and was refused arefund.” is elaborated with

“I am just wondering that since I am not going to have a tag on thistruck, is it possible to get the property tax refunded?”

“I am just wondering” has attribution to

“that” is the same unit as “is it possible to get the property taxrefunded?” which has condition “since I am not going to have a tag onthis truck”

As can be seen, the main subject of the topic is “Property tax on acar”. The question includes the contradiction: on one hand, allproperties are taxable, and on the other hand, the ownership is somewhatincomplete. A good response has to address both topic of the questionand clarify the inconsistency. To do that, the responder is making evenstronger claim concerning the necessity to pay tax on whatever is ownedirrespectively of the registration status. This example is a member ofpositive training set from our Yahoo! Answers evaluation domain. Themain subject of the topic is “Property tax on a car”. The questionincludes the contradiction: on one hand, all properties are taxable, andon the other hand, the ownership is somewhat incomplete. A goodanswer/response has to address both topic of the question and clarifythe inconsistency. The reader can observe that since the questionincludes rhetoric relation of contrast, the answer has to match it witha similar relation to be convincing. Otherwise, this answer would lookincomplete even to those who are not domain experts.

FIG. 8 depicts an exemplary response for the question represented inFIG. 7 , according to certain aspects of the present invention. Thecentral nucleus is “the property tax is assessed on property” elaboratedby “that you own”. “The property tax is assessed on property that youown” is also a nucleus elaborated by “Just because you chose to notregister it does not mean that you don't own it, so the tax is notrefundable. Even if you have not titled the vehicle yet, you still ownit within the boundaries of the tax district, so the tax is payable.Note that all states give you a limited amount of time to transfer titleand pay the use tax.”

The nucleus “The property tax is assessed on property that you own. Justbecause you chose to not register it does not mean that you don't ownit, so the tax is not refundable. Even if you have not titled thevehicle yet, you still own it within the boundaries of the tax district,so the tax is payable. Note that all states give you a limited amount oftime to transfer title and pay the use tax.” is elaborated by “therewill be penalties on top of the normal taxes and fees” with condition“If you apply late,” which in turn is elaborated by the contrast of “butyou absolutely need to title it within the period of time stipulated instate law.” and “You don't need to register it at the same time.”.

Comparing the DT of FIG. 7 and DT of FIG. 8 , enables a determination ofhow well matched the response (FIG. 8 ) is to the request (FIG. 7 ). Insome aspects of the present invention, the above framework is used, atleast in part, to determine the DTs for the request/response and therhetoric agreement between the DTs.

In another example, the question “What does The Investigative Committeeof the Russian Federation do” has at least two answers, for example, anofficial answer or an actual answer.

FIG. 9 illustrates a discourse tree for an official answer in accordancewith an aspect. As depicted in FIG. 9 , an official answer, or missionstatement states that “The Investigative Committee of the RussianFederation is the main federal investigating authority which operates asRussia's Anti-corruption agency and has statutory responsibility forinspecting the police forces, combating police corruption and policemisconduct, is responsible for conducting investigations into localauthorities and federal governmental bodies.”

FIG. 10 illustrates a discourse tree 1000 for a raw answer in accordancewith an aspect. As depicted in FIG. 10 , another, perhaps more honest,answer states that “Investigative Committee of the Russian Federation issupposed to fight corruption. However, top-rank officers of theInvestigative Committee of the Russian Federation are charged withcreation of a criminal community. Not only that, but their involvementin large bribes, money laundering, obstruction of justice, abuse ofpower, extortion, and racketeering has been reported. Due to theactivities of these officers, dozens of high-profile cases including theones against criminal lords had been ultimately ruined.”

The choice of answers depends on context. Rhetoric structure allowsdifferentiating between “official”, “politically correct”,template-based answers and “actual”, “raw”, “reports from the field”, or“controversial” answers. (See FIGS. 9 and 10 ). Sometimes, the questionitself can give a hint about which category of answers is expected. If aquestion is formulated as a factoid or definitional one, without asecond meaning, then the first category of answers is suitable.Otherwise, if a question has the meaning “tell me what it really is,”then the second category is appropriate. In general, after extracting arhetoric structure from a question, selecting a suitable answer thatwould have a similar, matching, or complementary rhetoric structure iseasier.

The official answer is based on elaboration and joints, which areneutral in terms of controversy a text might contain (See FIG. 9 ). Atthe same time, the row answer includes the contrast relation. Thisrelation is extracted between the phrase for what an agent is expectedto do and what this agent was discovered to have done.

Using Discourse Trees to Determine Informative Text for Indexing

As discussed, certain aspects use discourse trees to determineinformative portions of text to index. This approach increases theamount of text that is informative in answers to questions provided bysearch engines or autonomous agents and facilitates the generation ofadditional questions that can be answered by a body of text, therebymaking the text more accessible via question answering. Certain aspectsare described with respect to FIG. 11 for example purposes.

FIG. 11 illustrates a discourse tree for an example of an answer to aquestion, in accordance with an aspect. FIG. 11 illustrates discoursetree 1100, which represents an answer to the following question: “Howshould I plan to pay for taxes resulting from converting to a Roth IRA?”FIG. 11 also depicts elementary discourse units corresponding tofragments of the text as terminal nodes 1110-1125.

The answer corresponding to discourse tree 100 is: “[t]o help maximizeyour retirement savings, it's generally a good idea to consider notusing the proceeds from the conversion to pay the resulting tax costs.Instead, you should consider using cash or other savings held innonretirement accounts. Using retirement account funds to pay the taxeswill reduce the amount you would have available to potentially growtax-free in your new Roth IRA. Additionally, if you are under 59½, usingfunds from your retirement account could result in an additional 10% taxpenalty, which may significantly reduce the potential benefit ofconversion.”

The answer could be obtained from a source such as a Frequently AskedQuestions (FAQ) database or a question-answer index. A question-answerindex can include multiple questions and corresponding answers. But asfurther explained, some fragments in each answer are more informative toanswering a question than other fragments. For example, the phrase “itis generally a good idea” adds little to the answer, whereas “considernot using the proceeds from the conversion” is informative to the userwho posed the original question.

Additionally, each answer in the question-answer index may provideadditional questions that can be answered, which are in turn indexed,increasing the usefulness of the data. For example, “at what age do Ipay a penalty for using retirement funds?” could be answered by the text(e.g., “age 59½”). Certain aspects can determine informative text from abody of text and such additional questions that can be answered from thebody of text.

The nodes selected for indexing are nodes 1110, 1112, 1115, 1117, 1119,1123, and 1124 (shown with underlining). A general rule is that the EDUsrepresenting nuclei should be indexed, and the EDUs representingsatellites is discarded.

As depicted, a node that is the first, or upper, of two related nodes isa nucleus. The node that is the second, or lower, of two related nodes,is the satellite. Note that in a conditional relation, the “if” EDU isthe satellite. For example, phrases from the nucleus elementarydiscourse units and their respective nodes are:

-   -   help maximize your retirement savings (Node 1110)    -   proceeds from the conversion (Node 1112)    -   cash or other savings held in nonretirement accounts (Node 1115)    -   retirement account funds; (Node 1117)    -   using funds from your retirement account (Node 1123)    -   result in an additional 10% tax penalty (Node 1124)

The satellite EDU expressions are:

-   -   it's generally a good idea (Node 1111)    -   pay the resulting tax costs (Node 1118)    -   held in nonretirement accounts (Node 1116)    -   to pay the taxes (Node 1118)    -   you would have available to potentially (Node 1120)    -   if you are under 59½ . . . (Node 1122)

It can be seen that generally speaking the nodes of terminal nodes1100-1125 that are identified as nuclei are directly informative to thetopic of the answer and therefore should be indexed. In contrast, thesatellite nodes for a particular rhetorical relationship are notinformative to the question and therefore should not be indexed. Forexample, the phrase “it is generally a good idea” (node 1111) is notrelated to finance. The phrase “pay the resulting tax costs” (node 1118)is detached from the context of the question, as is “held innonretirement accounts” (node 1116). Additionally, “you would haveavailable to potentially” (node 1120) is a counterfactual question thatis unlikely to occur in a user question. Finally, “if you are under 59½”(node 1122) is a condition that is not necessary directly queried.

The above hypothesis that only EDUs that are nucleus of rhetoricrelations should be indexed and all satellite EDUs should not beselected for indexing is illustrated by the “elaboration” relationship.In the elaboration relationship, the nucleus expresses more importantinformation than satellite. A satellite may express a detail ofinformation being communicated that is unlikely to be cited by a userquery. For example, nodes 1124 and 1125 are related by an elaborationrelationship. A can be seen, “could result in an additional 10% penalty”is more important than “which may significantly reduce the potentialbenefit of conversion.” Accordingly, the satellite fragments of textshould not be matched with a potential question to deliver thisparticular answer.

Different Rhetoric Relations

But the general rule described above can be subject to certainexceptions. For example, under certain conditions, the “contrast” and“attribution” relations can require indexing of the satellite ratherthan the nucleus. Additionally, for the “same-unit” and “joint”relations, both the nucleus and the satellite are indexed. Differentrhetoric relations can have different rules, as shown in Table 4 below:

TABLE 4 Relation Example Indexing rule Elabor- To achieve some stateNucleus ation [nucleus] | do this and that [satellite] Enable- A querymay be of the form Nucleus ment “how to achieve some state?” but lesslikely be of the form “what can I achieve doing this and that?”. Condi-A query may be of the form When the question is of the type tion “how toachieve some state?” “when/where/under what but less likely be of theform condition . . . ”, index the if part “what can I achieve doing (thesatellite) this and that?”. Contrast Index the nucleus. The satelliteincludes facts which are unusual, unexpected, unanticipated. Attribu-General rule: index the nucleus tion (may occur in a factoid question).Do not index the satellite part on whom the factoid is attributed (as itis usually a minor detail). Exception: a query by an author. In thiscase, such queries texts should be transformed into a structured way andcovered by a different kind of search technology. Same- Index bothnucleus and satellite Unit because of the symmetric relationship ofsame-unit. Joint Index both nucleus and satellite because of thesymmetric relationship of joint.

FIG. 12 depicts a flowchart illustrating an example of a process fordetermining informative text for indexing, in accordance with an aspect.FIG. 12 is discussed with respect to FIG. 11 for example purposes.

At block 1201, process 1200 involves accessing a body of text includingfragments. Examples of bodies of text are user utterances, onlinecontent, stored electronic content, and scanned documents. Text caninclude question/answer pairs. In an example, indexing application 102accesses the text associated with discourse tree 1100.

At block 1202, process 1200 involves creating a discourse tree from thebody of text. Creating a discourse tree involves determining elementarydiscourse units and the rhetorical relations between each pair ofelementary discourse units. The discourse tree includes nodes, eachnonterminal node representing a rhetorical relationship between two ofthe fragments, and each terminal node is associated with one or morefragments and is associated with a non-terminal node. Continuing theexample, indexing application 102 creates discourse tree 1100. Anydiscourse tree parser can be used.

At block 1203, process 1200 involves identifying a rhetoricalrelationship for each non-terminal node. The rhetorical relationshipindicates the relationship of the two child nodes, which can be terminalnodes, or other non-terminal nodes. Continuing the example, indexingapplication 102 identifies various rhetorical relations as illustratedin FIG. 11 such as elaboration, same unit, enablement, etc. Eachterminal node 1100-1125 is associated with a rhetorical relation. Forexample, indexing application 102 identifies nodes 1112 and 1113 asrelated by an enablement relation.

At block 1204, process 1200 involves labeling each terminal nodeassociated with a non-terminal node as a nucleus or a satellite. UsingRhetorical Structure Theory, indexing application 102 labels eachterminal node as a nucleus or a satellite.

At block 1205, process 1200 involves accessing a rule associated withthe rhetorical relationship. The rule specifies accessing a fragment oftext associated with either the nucleus or the satellite. Continuing theexample, indexing application 102 accesses a rule for enablement. Asdescribed in Table 4, for EDUs associated with enablement, the nucleusis indexed. Indexing application 102 continues block 1204 for eachrhetorical relation.

At block 1206, process 1200 involves selecting, based on the rule,either (i) the fragment associated with the nucleus or the (ii) fragmentassociated with the satellite. Using the rule identified at block 1204,indexing application 102 selects the nucleus, node 1112, for indexing.Indexing application 102 continues block 1205 for each rhetoricalrelation.

At block 1207, process 1200 involves creating a searchable index withmultiple entries, each entry corresponding to a selected fragment andrepresenting a question that can be answered by the body of text.Continuing the example, indexing application 102 creates a searchableindex with all selected elementary discourse units gathered during block1205. Specifically, indexing application 102 gathers nodes 1110, 1112,1115, 1117, 1119, 1123, and 1124. In some cases, indexing application102 determines a question that can be answered by each indexed EDU andstores the question along with the indexed text in question-answer index135.

Building Additional Questions from Indexed Text

Indexing application 102 can use a multi-tiered approach to answeringquestions or queries. For example, indexing application 102 can use thequestion-answer index 135 (e.g., a Frequently-Asked Question database)or the entirety of the text from which question-answer index 135 wasconstructed. In the example that follows, indexing application 102 canfirst check question-answer index 135 for a suitable match, then checkthe list of alternative questions, i.e., the searchable index determinedby process 1200, then search all of the answers in question-answer index135.

FIG. 13 depicts a flowchart illustrating an example of a process forresponding to searches, in accordance with an aspect. FIG. 13 can usequestion-answer index 135, a searchable index that comprises informativefragments of text determined by process 1200, or anotherpreviously-indexed text.

At block 1301, process 1300 involves creating an index of additionalquestions by determining, for each entry of the searchable index, anadditional question that is answered by the entry. Different methods canbe used to determine questions from a particular informative fragment oftext.

In an example, indexing application 102 receives an arbitrary sentenceand outputs a set of questions. Indexing application 102 builds a parsetree. Indexing application 102 selects nodes of the parse tree thatrepresent nouns, verbs, and adjectives. Next, for each selected node,indexing application 102 forms a reduction of a parse tree by removingthe particular node. Indexing application 102 builds a question for thereduction by substituting a Wh word (what, who, where, whose, why, etc.)for the removed node. Indexing application 102 selects a proper Wh wordfollowing a set of rules. In an example, rules include substituting“who” or “what” for a noun, “what . . . do,” and “which way” or “how is”for an adjective.

FIG. 14 depicts a parse tree for a sentence used to illustrate forming aquestion from text, in accordance with an aspect. FIG. 14 depicts aparse tree 1400 for the sentence “Joe packed his tools and materialsneatly.” As depicted, parse tree 1400 includes 11 nodes, 7 of which areterminal nodes. Table 5 below illustrates questions generated bydeleting odd-numbered nodes in the sentence using the method described.

TABLE 5 Node deleted Question Generated 3 What did Joe pack neatly? 5Who packed his tools and materials neatly? 7 Whose tools and materialsdid Joe pack neatly? 9 No question: Structure words are not deleted. 11How did Joe pack his tools and materials?

Returning to FIG. 13 , at block 1302, process 1300 involves receiving aquery from an external device such as mobile device 170. A query can bean input to an autonomous agent (ChatBot), or a search provided by auser.

At block 1303, process 1300 involves searching, in each question of thequestion-answer index for a match corresponding to the query. Thequestion-answer pairs can be a FAQ, e.g., with a question andcorresponding answer.

At block 1304, process 1300 involves responsive to determining thatfewer than a threshold number of matches are available, searching foradditional matches to the query in the index of additional questions.The searchable index can be generated, for example, by process 1200. Byusing the searchable index that contains only text fragments informativeto the answer, indexing application 102 need not parse through andpotentially output less-informative text to the question.

Continuing the example, index application 102 determines that less thana threshold number of matches to the query are available in the index ofadditional questions. Example thresholds are a percentage of the totalavailable questions, a predetermined number, or a number of questionsdetermined by a particular domain, subject, or system configuration. Inresponse, indexing application 102 searches the question-answer index135.

At block 1305, process 1300 involves, responsive to determining thatfewer than a threshold number of additional matches are available in theindex of additional questions, searching for further matches in eachanswer of the an index of original answers. The threshold used at block1305 can be different from the threshold used at block 1304. Continuingthe example, indexing application 102 searches for a match in eachanswer of the question-answer index.

At block 1306, process 1300 involves providing the response to theexternal device. Example external devices include mobile device 170 orremote devices connected across data network 104. Alternatively,indexing application 102 can provide the answer directly to a user.

Autonomous Agents

As discussed, certain aspects described herein enable improvedautonomous agents (ChatBots). A conversation between Human A and Human Bis a form of discourse. For example, applications exist such asFaceBook® Messenger, WhatsApp®, Slack,® SMS, etc., a conversationbetween A and B may typically be via messages in addition to moretraditional email and voice conversations. A ChatBot (which may also becalled intelligent bots or virtual assistant, etc.) is an “intelligent”machine that, for example, replaces human B and to various degreesmimics the conversation between two humans. An example ultimate goal isthat human A cannot tell whether B is a human or a machine (the Turningtest, developed by Alan Turing in 1950). Discourse analysis, artificialintelligence, including machine learning, and natural languageprocessing, have made great strides toward the long-term goal of passingthe Turing test. Of course, with computers being more and more capableof searching and processing vast repositories of data and performingcomplex analysis on the data to include predictive analysis, thelong-term goal is the ChatBot being human-like and a computer combined.

For example, users can interact with the Intelligent Bots Platformthrough a conversational interaction. This interaction, also called theconversational user interface (UI), is a dialog between the end user andthe ChatBot, just as between two human beings. It could be as simple asthe end user saying “Hello” to the ChatBot and the ChatBot respondingwith a “Hi” and asking the user how it can help, or it could be atransactional interaction in a banking ChatBot, such as transferringmoney from one account to the other, or an informational interaction ina HR ChatBot, such as checking for vacation balance, or asking an FAQ ina retail ChatBot, such as how to handle returns. Natural languageprocessing (NLP) and machine learning (ML) algorithms combined withother approaches can be used to classify end user intent. An intent at ahigh level is what the end user would like to accomplish (e.g., getaccount balance, make a purchase). An intent is essentially, a mappingof customer input to a unit of work that the backend should perform.Therefore, based on the phrases uttered by the user in the ChatBot,these are mapped that to a specific and discrete use case or unit ofwork, for e.g. check balance, transfer money and track spending are all“use cases” that the ChatBot should support and be able to work outwhich unit of work should be triggered from the free text entry that theend user types in a natural language.

The underlying rational for having a ChatBot respond like a human isthat the human brain can formulate and understand the request and thengive a good response to the human request much better than a machine.Thus, there should be significant improvement in the request/response ofa ChatBot, if human B is mimicked. So an initial part of the problem ishow does the human brain formulate and understand the request? To mimic,a model is used. RST and DT allow a formal and repeatable way of doingthis.

At a high level, there are typically two types of requests: (1) Arequest to perform some action; and (2) a request for information, e.g.,a question. The first type has a response in which a unit of work iscreated. The second type has a response that is, e.g., a good answer, tothe question. The answer could take the form of, for example, in someaspects, the AI constructing an answer from its extensive knowledgebase(s) or from matching the best existing answer from searching theinternet or intranet or other publically/privately available datasources.

FIG. 15 illustrates an autonomous agent answering user questions inaccordance with an aspect. FIG. 15 depicts chat 1500, user messages 1501and 1503, and agent responses 1502 and 1504. Agent responses 1502 and1504 can be provided by indexing application 102. As depicted, the usermessage 1501 asks “at what age do I normally pay a penalty for usingretirement funds?” Indexing application 102 identifies the answer fromthe body of text indexed using process 1200 and provides agent response1502, which states “age 59½.” User asks a follow on question 1503 “Whatis the penalty?” In response, the agent answers “An additional 10% taxpenalty.”

The features depicted in FIG. 15 can be implemented by computing device101 or by indexing application 102.

Using Classifiers to Determine Text for Indexing

Certain aspects use machine learning models to learn rules such as thosedepicted in Table 4. A machine learning problem can be formulated as aclassification problem that classifies EDUs into a first class that issuitable for indexing (i.e., informative) and forming alternativequestions for an answer and a second class that is not suitable forindexing (i.e., not informative).

In an example, a training data set is formed by using text gathered froma search engine. For example, search answer popularity, or search rank,can be learned by a search engine on the basis of a high number ofsearches for the same query and user selection.

To accumulate question-answer pairs with marked answers, a selection ofqueries against short texts can be run. The portions of these texts thatare used for matching can be identified. Because longer queries arenecessary to assure a corresponding match is nontrivial, publicquestion-answer pages such as Yahoo! Answers datasets can be used. Morespecifically, questions from such datasets can be formed from a firstsentence of the dataset. In an example, Microsoft Cognitive Services(Bing Search engine API) can execute these queries. Search results whichare short texts (4-6 sentences) are selected as such texts suitable forparsing and discourse analysis.

Matched fragments of these texts are taken as elements of the trainingset. Such fragments from the top ten or more pages of search resultforms a positive dataset, i.e. informative fragments. It includes thefragments of texts considered by the search engine to be of highrelevance. For the negative dataset, fragments with matched keywordsfrom the set of lower ranked (100-1000+) search results pages are taken,as these results are assumed to be less relevant.

FIG. 16 depicts a flowchart illustrating an example of a process fortraining a classification model to determine informative text forindexing, in accordance with an aspect. An example classification modelis machine-learning model 120, which can use different models such asclassifiers, Tree Kernels, or Support Vector Machine, or SVM TK.

At block 1601, process 1600 involves accessing a set of training datacomprising a set of training pairs; each training data pair includestext and an expected classification, the set of training data includingboth (i) a first training data pair that has an expected classificationof a nucleus and (ii) a second training data pair that has an expectedclassification of a satellite.

At block 1602, process 1600 involves providing one of the training datapairs to the classification model. Accordingly, the classification modelreceives a body of text and an expected classification.

At block 1603, process 1600 involves receiving a determinedclassification from the classification model.

At block 1604, process 1600 involves calculating a loss function bycalculating a difference between the determined classification and theexpected classification. Different loss functions are possible such asmean-square error, likelihood loss, log (or cross entropy) loss, etc.

At block 1605, process 1600 involves adjusting internal parameters ofthe classification model to minimize the loss function. In this manner,the classification model learns to improve the accuracy of itspredictions.

At block 1606, process 1600 involves using the trained classificationmodel. For example, the trained classification model can be used inprocesses 1200 or 1300. More specifically, the trained classificationmodel can be used in place of the rule-based scheme discussed withrespect to Table 4 and implemented in process 1200 or 1300.

FIG. 17 depicts a simplified diagram of a distributed system 1700 forimplementing one of the aspects. In the illustrated aspect, distributedsystem 1700 includes one or more client computing devices 1702, 1704,1706, and 1708, which are configured to execute and operate a clientapplication such as a web browser, proprietary client (e.g., OracleForms), or the like over one or more network(s) 1710. Server 1712 may becommunicatively coupled with remote client computing devices 1702, 1704,1706, and 1708 via network 1710.

In various aspects, server 1712 may be adapted to run one or moreservices or software applications provided by one or more of thecomponents of the system. The services or software applications caninclude non-virtual and virtual environments. Virtual environments caninclude those used for virtual events, tradeshows, simulators,classrooms, shopping exchanges, and enterprises, whether two- orthree-dimensional (3D) representations, page-based logical environments,or otherwise. In some aspects, these services may be offered asweb-based or cloud services or under a Software as a Service (SaaS)model to the users of client computing devices 1702, 1704, 1706, and/or1708. Users operating client computing devices 1702, 1704, 1706, and/or1708 may in turn utilize one or more client applications to interactwith server 1712 to utilize the services provided by these components.

In the configuration depicted in the figure, the software components1718, 1720 and 1722 of distributed system 1700 are shown as beingimplemented on server 1712. In other aspects, one or more of thecomponents of distributed system 1700 and/or the services provided bythese components may also be implemented by one or more of the clientcomputing devices 1702, 1704, 1706, and/or 1708. Users operating theclient computing devices may then utilize one or more clientapplications to use the services provided by these components. Thesecomponents may be implemented in hardware, firmware, software, orcombinations thereof. It should be appreciated that various differentsystem configurations are possible, which may be different fromdistributed system 1700. The aspect shown in the figure is thus oneexample of a distributed system for implementing an aspect system and isnot intended to be limiting.

Client computing devices 1702, 1704, 1706, and/or 1708 may be portablehandheld devices (e.g., an iPhone®, cellular telephone, an iPad®,computing tablet, a personal digital assistant (PDA)) or wearabledevices (e.g., a Google Glass® head mounted display), running softwaresuch as Microsoft Windows Mobile®, and/or a variety of mobile operatingsystems such as iOS, Windows Phone, Android, BlackBerry 10, Palm OS, andthe like, and being Internet, e-mail, short message service (SMS),Blackberry®, or other communication protocol enabled. The clientcomputing devices can be general purpose personal computers including,by way of example, personal computers and/or laptop computers runningvarious versions of Microsoft Windows®, Apple Macintosh®, and/or Linuxoperating systems. The client computing devices can be workstationcomputers running any of a variety of commercially-available UNIX® orUNIX-like operating systems, including without limitation the variety ofGNU/Linux operating systems, such as for example, Google Chrome OS.Alternatively, or in addition, client computing devices 1702, 1704,1706, and 1708 may be any other electronic device, such as a thin-clientcomputer, an Internet-enabled gaming system (e.g., a Microsoft Xboxgaming console with or without a Kinect® gesture input device), and/or apersonal messaging device, capable of communicating over network(s)1710.

Although exemplary distributed system 1700 is shown with four clientcomputing devices, any number of client computing devices may besupported. Other devices, such as devices with sensors, etc., mayinteract with server 1712.

Network(s) 1710 in distributed system 1700 may be any type of networkfamiliar to those skilled in the art that can support datacommunications using any of a variety of commercially-availableprotocols, including without limitation TCP/IP (transmission controlprotocol/Internet protocol), SNA (systems network architecture), IPX(Internet packet exchange), AppleTalk, and the like. Merely by way ofexample, network(s) 1710 can be a local area network (LAN), such as onebased on Ethernet, Token-Ring and/or the like. Network(s) 1710 can be awide-area network and the Internet. It can include a virtual network,including without limitation a virtual private network (VPN), anintranet, an extranet, a public switched telephone network (PSTN), aninfra-red network, a wireless network (e.g., a network operating underany of the Institute of Electrical and Electronics (IEEE) 802.17 suiteof protocols, Bluetooth®, and/or any other wireless protocol); and/orany combination of these and/or other networks.

Server 1712 may be composed of one or more general purpose computers,specialized server computers (including, by way of example, PC (personalcomputer) servers, UNIX® servers, mid-range servers, mainframecomputers, rack-mounted servers, etc.), server farms, server clusters,or any other appropriate arrangement and/or combination. Server 1712 caninclude one or more virtual machines running virtual operating systems,or other computing architectures involving virtualization. One or moreflexible pools of logical storage devices can be virtualized to maintainvirtual storage devices for the server. Virtual networks can becontrolled by server 1712 using software defined networking. In variousaspects, server 1712 may be adapted to run one or more services orsoftware applications described in the foregoing disclosure. Forexample, server 1712 may correspond to a server for performingprocessing described above according to an aspect of the presentdisclosure.

Server 1712 may run an operating system including any of those discussedabove, as well as any commercially available server operating system.Server 1712 may also run any of a variety of additional serverapplications and/or mid-tier applications, including HTTP (hypertexttransport protocol) servers, FTP (file transfer protocol) servers, CGI(common gateway interface) servers, JAVA® servers, database servers, andthe like. Exemplary database servers include without limitation thosecommercially available from Oracle, Microsoft, Sybase, IBM(International Business Machines), and the like.

In some implementations, server 1712 may include one or moreapplications to analyze and consolidate data feeds and/or event updatesreceived from users of client computing devices 1702, 1704, 1706, and1708. As an example, data feeds and/or event updates may include, butare not limited to, Twitter® feeds, Facebook® updates or real-timeupdates received from one or more third party information sources andcontinuous data streams, which may include real-time events related tosensor data applications, financial tickers, network performancemeasuring tools (e.g., network monitoring and traffic managementapplications), clickstream analysis tools, automobile trafficmonitoring, and the like. Server 1712 may also include one or moreapplications to display the data feeds and/or real-time events via oneor more display devices of client computing devices 1702, 1704, 1706,and 1708.

Distributed system 1700 may also include one or more databases 1714 and1716. Databases 1714 and 1716 may reside in a variety of locations. Byway of example, one or more of databases 1714 and 1716 may reside on anon-transitory storage medium local to (and/or resident in) server 1712.Alternatively, databases 1714 and 1716 may be remote from server 1712and in communication with server 1712 via a network-based or dedicatedconnection. In one set of aspects, databases 1714 and 1716 may reside ina storage-area network (SAN). Similarly, any necessary files forperforming the functions attributed to server 1712 may be stored locallyon server 1712 and/or remotely, as appropriate. In one set of aspects,databases 1714 and 1716 may include relational databases, such asdatabases provided by Oracle, that are adapted to store, update, andretrieve data in response to SQL-formatted commands.

FIG. 18 is a simplified block diagram of one or more components of asystem environment 1800 by which services provided by one or morecomponents of an aspect system may be offered as cloud services inaccordance with an aspect of the present disclosure. In the illustratedaspect, system environment 1800 includes one or more client computingdevices 1804, 1806, and 1808 that may be used by users to interact witha cloud infrastructure system 1802 that provides cloud services. Theclient computing devices may be configured to operate a clientapplication such as a web browser, a proprietary client application(e.g., Oracle Forms), or some other application, which may be used by auser of the client computing device to interact with cloudinfrastructure system 1802 to use services provided by cloudinfrastructure system 1802.

It should be appreciated that cloud infrastructure system 1802 depictedin the figure may have other components than those depicted. Further,the aspect shown in the figure is only one example of a cloudinfrastructure system that may incorporate an aspect of the invention.In some other aspects, cloud infrastructure system 1802 may have more orfewer components than shown in the figure, may combine two or morecomponents, or may have a different configuration or arrangement ofcomponents.

Client devices 1804, 1806, and 1808 may be devices similar to thosedescribed above for 2802, 2804, 2806, and 2808.

Although exemplary system environment 1800 is shown with three clientcomputing devices, any number of client computing devices may besupported. Other devices such as devices with sensors, etc. may interactwith cloud infrastructure system 1802.

Network(s) 1810 may facilitate communications and exchange of databetween client devices 1804, 1806, and 1808 and cloud infrastructuresystem 1802. Each network may be any type of network familiar to thoseskilled in the art that can support data communications using any of avariety of commercially-available protocols, including those describedabove for network(s) 1810.

Cloud infrastructure system 1802 may comprise one or more computersand/or servers that may include those described above for server 1712.

In certain aspects, services provided by the cloud infrastructure systemmay include a host of services that are made available to users of thecloud infrastructure system on demand, such as online data storage andbackup solutions, Web-based e-mail services, hosted office suites anddocument collaboration services, database processing, managed technicalsupport services, and the like. Services provided by the cloudinfrastructure system can dynamically scale to meet the needs of itsusers. A specific instantiation of a service provided by cloudinfrastructure system is referred to herein as a “service instance.” Ingeneral, any service made available to a user via a communicationnetwork, such as the Internet, from a cloud service provider's system isreferred to as a “cloud service.” Typically, in a public cloudenvironment, servers and systems that make up the cloud serviceprovider's system are different from the customer's own on-premisesservers and systems. For example, a cloud service provider's system mayhost an application, and a user may, via a communication network such asthe Internet, on demand, order and use the application.

In some examples, a service in a computer network cloud infrastructuremay include protected computer network access to storage, a hosteddatabase, a hosted web server, a software application, or other serviceprovided by a cloud vendor to a user, or as otherwise known in the art.For example, a service can include password-protected access to remotestorage on the cloud through the Internet. As another example, a servicecan include a web service-based hosted relational database and ascript-language middleware engine for private use by a networkeddeveloper. As another example, a service can include access to an emailsoftware application hosted on a cloud vendor's web site.

In certain aspects, cloud infrastructure system 1802 may include a suiteof applications, middleware, and database service offerings that aredelivered to a customer in a self-service, subscription-based,elastically scalable, reliable, highly available, and secure manner. Anexample of such a cloud infrastructure system is the Oracle Public Cloudprovided by the present assignee.

Large volumes of data, sometimes referred to as big data, can be hostedand/or manipulated by the infrastructure system on many levels and atdifferent scales. Such data can include data sets that are so large andcomplex that it can be difficult to process using typical databasemanagement tools or traditional data processing applications. Forexample, terabytes of data may be difficult to store, retrieve, andprocess using personal computers or their rack-based counterparts. Suchsizes of data can be difficult to work with using most currentrelational database management systems and desktop statistics andvisualization packages. They can require massively parallel processingsoftware running thousands of server computers, beyond the structure ofcommonly used software tools, to capture, curate, manage, and processthe data within a tolerable elapsed time.

Extremely large data sets can be stored and manipulated by analysts andresearchers to visualize large amounts of data, detect trends, and/orotherwise interact with the data. Tens, hundreds, or thousands ofprocessors linked in parallel can act upon such data in order to presentit or simulate external forces on the data or what it represents. Thesedata sets can involve structured data, such as that organized in adatabase or otherwise according to a structured model, and/orunstructured data (e.g., emails, images, data blobs (binary largeobjects), web pages, complex event processing). By leveraging an abilityof an aspect to relatively quickly focus more (or fewer) computingresources upon an objective, the cloud infrastructure system may bebetter available to carry out tasks on large data sets based on demandfrom a business, government agency, research organization, privateindividual, group of like-minded individuals or organizations, or otherentity.

In various aspects, cloud infrastructure system 1802 may be adapted toautomatically provision, manage and track a customer's subscription toservices offered by cloud infrastructure system 1802. Cloudinfrastructure system 1802 may provide the cloud services via differentdeployment models. For example, services may be provided under a publiccloud model in which cloud infrastructure system 1802 is owned by anorganization selling cloud services (e.g., owned by Oracle) and theservices are made available to the general public or different industryenterprises. As another example, services may be provided under aprivate cloud model in which cloud infrastructure system 1802 isoperated solely for a single organization and may provide services forone or more entities within the organization. The cloud services mayalso be provided under a community cloud model in which cloudinfrastructure system 1802 and the services provided by cloudinfrastructure system 1802 are shared by several organizations in arelated community. The cloud services may also be provided under ahybrid cloud model, which is a combination of two or more differentmodels.

In some aspects, the services provided by cloud infrastructure system1802 may include one or more services provided under Software as aService (SaaS) category, Platform as a Service (PaaS) category,Infrastructure as a Service (IaaS) category, or other categories ofservices including hybrid services. A customer, via a subscriptionorder, may order one or more services provided by cloud infrastructuresystem 1802. Cloud infrastructure system 1802 then performs processingto provide the services in the customer's subscription order.

In some aspects, the services provided by cloud infrastructure system1802 may include, without limitation, application services, platformservices and infrastructure services. In some examples, applicationservices may be provided by the cloud infrastructure system via a SaaSplatform. The SaaS platform may be configured to provide cloud servicesthat fall under the SaaS category. For example, the SaaS platform mayprovide capabilities to build and deliver a suite of on-demandapplications on an integrated development and deployment platform. TheSaaS platform may manage and control the underlying software andinfrastructure for providing the SaaS services. By utilizing theservices provided by the SaaS platform, customers can utilizeapplications executing on the cloud infrastructure system. Customers canacquire the application services without the need for customers topurchase separate licenses and support. Various different SaaS servicesmay be provided. Examples include, without limitation, services thatprovide solutions for sales performance management, enterpriseintegration, and business flexibility for large organizations.

In some aspects, platform services may be provided by the cloudinfrastructure system via a PaaS platform. The PaaS platform may beconfigured to provide cloud services that fall under the PaaS category.Examples of platform services may include without limitation servicesthat enable organizations (such as Oracle) to consolidate existingapplications on a shared, common architecture, as well as the ability tobuild new applications that leverage the shared services provided by theplatform. The PaaS platform may manage and control the underlyingsoftware and infrastructure for providing the PaaS services. Customerscan acquire the PaaS services provided by the cloud infrastructuresystem without the need for customers to purchase separate licenses andsupport. Examples of platform services include, without limitation,Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS),and others.

By utilizing the services provided by the PaaS platform, customers canemploy programming languages and tools supported by the cloudinfrastructure system and also control the deployed services. In someaspects, platform services provided by the cloud infrastructure systemmay include database cloud services, middleware cloud services (e.g.,Oracle Fusion Middleware services), and Java cloud services. In oneaspect, database cloud services may support shared service deploymentmodels that enable organizations to pool database resources and offercustomers a Database as a Service in the form of a database cloud.Middleware cloud services may provide a platform for customers todevelop and deploy various business applications, and Java cloudservices may provide a platform for customers to deploy Javaapplications, in the cloud infrastructure system.

Various different infrastructure services may be provided by an IaaSplatform in the cloud infrastructure system. The infrastructure servicesfacilitate the management and control of the underlying computingresources, such as storage, networks, and other fundamental computingresources for customers utilizing services provided by the SaaS platformand the PaaS platform.

In certain aspects, cloud infrastructure system 1802 may also includeinfrastructure resources 1830 for providing the resources used toprovide various services to customers of the cloud infrastructuresystem. In one aspect, infrastructure resources 1830 may includepre-integrated and optimized combinations of hardware, such as servers,storage, and networking resources to execute the services provided bythe PaaS platform and the SaaS platform.

In some aspects, resources in cloud infrastructure system 1802 may beshared by multiple users and dynamically re-allocated per demand.Additionally, resources may be allocated to users in different timezones. For example, cloud infrastructure system 1830 may enable a firstset of users in a first time zone to utilize resources of the cloudinfrastructure system for a specified number of hours and then enablethe re-allocation of the same resources to another set of users locatedin a different time zone, thereby maximizing the utilization ofresources.

In certain aspects, a number of internal shared services 1832 may beprovided that are shared by different components or modules of cloudinfrastructure system 1802 and by the services provided by cloudinfrastructure system 1802. These internal shared services may include,without limitation, a security and identity service, an integrationservice, an enterprise repository service, an enterprise managerservice, a virus scanning and white list service, a high availability,backup and recovery service, service for enabling cloud support, anemail service, a notification service, a file transfer service, and thelike.

In certain aspects, cloud infrastructure system 1802 may providecomprehensive management of cloud services (e.g., SaaS, PaaS, and IaaSservices) in the cloud infrastructure system. In one aspect, cloudmanagement functionality may include capabilities for provisioning,managing and tracking a customer's subscription received by cloudinfrastructure system 1802, and the like.

In one aspect, as depicted in the figure, cloud management functionalitymay be provided by one or more modules, such as an order managementmodule 1820, an order orchestration module 1822, an order provisioningmodule 1824, an order management and monitoring module 1826, and anidentity management module 1828. These modules may include or beprovided using one or more computers and/or servers, which may begeneral purpose computers, specialized server computers, server farms,server clusters, or any other appropriate arrangement and/orcombination.

In exemplary operation 1834, a customer using a client device, such asclient device 1804, 1806 or 1808, may interact with cloud infrastructuresystem 1802 by requesting one or more services provided by cloudinfrastructure system 1802 and placing an order for a subscription forone or more services offered by cloud infrastructure system 1802. Incertain aspects, the customer may access a cloud User Interface (UI),cloud UI 1818, cloud UI 1814 and/or cloud UI 1816 and place asubscription order via these UIs. The order information received bycloud infrastructure system 1802 in response to the customer placing anorder may include information identifying the customer and one or moreservices offered by the cloud infrastructure system 1802 that thecustomer intends to subscribe to.

After an order has been placed by the customer, the order information isreceived via the cloud UIs, 1818, 1814 and/or 1816.

At operation 1836, the order is stored in order database 1818. Orderdatabase 1818 can be one of several databases operated by cloudinfrastructure system 1802 and operated in conjunction with other systemelements.

At operation 1838, the order information is forwarded to an ordermanagement module 1820. In some instances, order management module 1820may be configured to perform billing and accounting functions related tothe order, such as verifying the order, and upon verification, bookingthe order.

At operation 1840, information regarding the order is communicated to anorder orchestration module 1822. Order orchestration module 1822 mayutilize the order information to orchestrate the provisioning ofservices and resources for the order placed by the customer. In someinstances, order orchestration module 1822 may orchestrate theprovisioning of resources to support the subscribed services using theservices of order provisioning module 1824.

In certain aspects, order orchestration module 1822 enables themanagement of business processes associated with each order and appliesbusiness logic to determine whether an order should proceed toprovisioning. At operation 1842, upon receiving an order for a newsubscription, order orchestration module 1822 sends a request to orderprovisioning module 1824 to allocate resources and configure thoseresources needed to fulfill the subscription order. Order provisioningmodule 1824 enables the allocation of resources for the services orderedby the customer. Order provisioning module 1824 provides a level ofabstraction between the cloud services provided by cloud infrastructuresystem 1802 and the physical implementation layer that is used toprovision the resources for providing the requested services. Orderorchestration module 1822 may thus be isolated from implementationdetails, such as whether or not services and resources are actuallyprovisioned on the fly or pre-provisioned and only allocated/assignedupon request.

At operation 1844, once the services and resources are provisioned, anotification of the provided service may be sent to customers on clientdevices 1804, 1806 and/or 1808 by order provisioning module 1824 ofcloud infrastructure system 1802.

At operation 1846, the customer's subscription order may be managed andtracked by an order management and monitoring module 1826. In someinstances, order management and monitoring module 1826 may be configuredto collect usage statistics for the services in the subscription order,such as the amount of storage used, the amount data transferred, thenumber of users, and the amount of system up time and system down time.

In certain aspects, cloud infrastructure system 1802 may include anidentity management module 1828. Identity management module 1828 may beconfigured to provide identity services, such as access management andauthorization services in cloud infrastructure system 1802. In someaspects, identity management module 1828 may control information aboutcustomers who wish to utilize the services provided by cloudinfrastructure system 1802. Such information can include informationthat authenticates the identities of such customers and information thatdescribes which actions those customers are authorized to performrelative to various system resources (e.g., files, directories,applications, communication ports, memory segments, etc.). Identitymanagement module 1828 may also include the management of descriptiveinformation about each customer and about how and by whom thatdescriptive information can be accessed and modified.

FIG. 19 illustrates an exemplary computer system 1900, in which variousaspects of the present invention may be implemented. The system 1900 maybe used to implement any of the computer systems described above. Asshown in the figure, computer system 1900 includes a processing unit1904 that communicates with a number of peripheral subsystems via a bussubsystem 1902. These peripheral subsystems may include a processingacceleration unit 1906, an I/O subsystem 1908, a storage subsystem 1918and a communications subsystem 1924. Storage subsystem 1918 includestangible computer-readable storage media 1922 and a system memory 1910.

Bus subsystem 1902 provides a mechanism for letting the variouscomponents and subsystems of computer system 1900 communicate with eachother as intended. Although bus subsystem 1902 is shown schematically asa single bus, alternative aspects of the bus subsystem may utilizemultiple buses. Bus subsystem 1902 may be any of several types of busstructures including a memory bus or memory controller, a peripheralbus, and a local bus using any of a variety of bus architectures. Forexample, such architectures may include an Industry StandardArchitecture (ISA) bus, Micro Channel Architecture (MCA) bus, EnhancedISA (EISA) bus, Video Electronics Standards Association (VESA) localbus, and Peripheral Component Interconnect (PCI) bus, which can beimplemented as a Mezzanine bus manufactured to the IEEE P1986.1standard.

Processing unit 1904, which can be implemented as one or more integratedcircuits (e.g., a conventional microprocessor or microcontroller),controls the operation of computer system 1900. One or more processorsmay be included in processing unit 1904. These processors may includesingle core or multicore processors. In certain aspects, processing unit1904 may be implemented as one or more independent processing units 1932and/or 1934 with single or multicore processors included in eachprocessing unit. In other aspects, processing unit 1904 may also beimplemented as a quad-core processing unit formed by integrating twodual-core processors into a single chip.

In various aspects, processing unit 1904 can execute a variety ofprograms in response to program code and can maintain multipleconcurrently executing programs or processes. At any given time, some orall of the program code to be executed can be resident in processor(s)1904 and/or in storage subsystem 1918. Through suitable programming,processor(s) 1904 can provide various functionalities described above.Computer system 1900 may additionally include a processing accelerationunit 1906, which can include a digital signal processor (DSP), aspecial-purpose processor, and/or the like.

I/O subsystem 1908 may include user interface input devices and userinterface output devices. User interface input devices may include akeyboard, pointing devices such as a mouse or trackball, a touchpad ortouch screen incorporated into a display, a scroll wheel, a click wheel,a dial, a button, a switch, a keypad, audio input devices with voicecommand recognition systems, microphones, and other types of inputdevices. User interface input devices may include, for example, motionsensing and/or gesture recognition devices such as the Microsoft Kinect®motion sensor that enables users to control and interact with an inputdevice, such as the Microsoft Xbox® 360 game controller, through anatural user interface using gestures and spoken commands. Userinterface input devices may also include eye gesture recognition devicessuch as the Google Glass® blink detector that detects eye activity(e.g., ‘blinking’ while taking pictures and/or making a menu selection)from users and transforms the eye gestures as input into an input device(e.g., Google Glass®). Additionally, user interface input devices mayinclude voice recognition sensing devices that enable users to interactwith voice recognition systems (e.g., Siri® navigator), through voicecommands.

User interface input devices may also include, without limitation, threedimensional (3D) mice, joysticks or pointing sticks, gamepads andgraphic tablets, and audio/visual devices such as speakers, digitalcameras, digital camcorders, portable media players, webcams, imagescanners, fingerprint scanners, barcode reader 3D scanners, 3D printers,laser rangefinders, and eye gaze tracking devices. Additionally, userinterface input devices may include, for example, medical imaging inputdevices such as computed tomography, magnetic resonance imaging,position emission tomography, medical ultrasonography devices. Userinterface input devices may also include, for example, audio inputdevices such as MIDI keyboards, digital musical instruments and thelike.

User interface output devices may include a display subsystem, indicatorlights, or non-visual displays such as audio output devices, etc. Thedisplay subsystem may be a cathode ray tube (CRT), a flat-panel device,such as that using a liquid crystal display (LCD) or plasma display, aprojection device, a touch screen, and the like. In general, use of theterm “output device” is intended to include all possible types ofdevices and mechanisms for outputting information from computer system1900 to a user or other computer. For example, user interface outputdevices may include, without limitation, a variety of display devicesthat visually convey text, graphics and audio/video information such asmonitors, printers, speakers, headphones, automotive navigation systems,plotters, voice output devices, and modems.

Computer system 1900 may comprise a storage subsystem 1918 thatcomprises software elements, shown as being currently located within asystem memory 1910. System memory 1910 may store program instructionsthat are loadable and executable on processing unit 1904, as well asdata generated during the execution of these programs.

Depending on the configuration and type of computer system 1900, systemmemory 1910 may be volatile (such as random access memory (RAM)) and/ornon-volatile (such as read-only memory (ROM), flash memory, etc.) TheRAM typically contains data and/or program modules that are immediatelyaccessible to and/or presently being operated and executed by processingunit 1904. In some implementations, system memory 1910 may includemultiple different types of memory, such as static random access memory(SRAM) or dynamic random access memory (DRAM). In some implementations,a basic input/output system (BIOS), containing the basic routines thathelp to transfer information between elements within computer system1900, such as during start-up, may typically be stored in the ROM. Byway of example, and not limitation, system memory 1910 also illustratesapplication programs 1912, which may include client applications, Webbrowsers, mid-tier applications, relational database management systems(RDBMS), etc., program data 1914, and an operating system 1916. By wayof example, operating system 1916 may include various versions ofMicrosoft Windows®, Apple Macintosh®, and/or Linux operating systems, avariety of commercially-available UNIX® or UNIX-like operating systems(including without limitation the variety of GNU/Linux operatingsystems, the Google Chrome® OS, and the like) and/or mobile operatingsystems such as iOS, Windows® Phone, Android® OS, BlackBerry® 10 OS, andPalm® OS operating systems.

Storage subsystem 1918 may also provide a tangible computer-readablestorage medium for storing the basic programming and data constructsthat provide the functionality of some aspects. Software (programs, codemodules, instructions) that when executed by a processor provide thefunctionality described above may be stored in storage subsystem 1918.These software modules or instructions may be executed by processingunit 1904. Storage subsystem 1918 may also provide a repository forstoring data used in accordance with the present invention.

Storage subsystem 1918 may also include a computer-readable storagemedia reader 1920 that can further be connected to computer-readablestorage media 1922. Together and, optionally, in combination with systemmemory 1910, computer-readable storage media 1922 may comprehensivelyrepresent remote, local, fixed, and/or removable storage devices plusstorage media for temporarily and/or more permanently containing,storing, transmitting, and retrieving computer-readable information.

Computer-readable storage media 1922 containing code, or portions ofcode, can also include any appropriate media known or used in the art,including storage media and communication media, such as but not limitedto, volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage and/or transmissionof information. This can include tangible, non-transitorycomputer-readable storage media such as RAM, ROM, electronicallyerasable programmable ROM (EEPROM), flash memory or other memorytechnology, CD-ROM, digital versatile disk (DVD), or other opticalstorage, magnetic cassettes, magnetic tape, magnetic disk storage orother magnetic storage devices, or other tangible computer readablemedia. When specified, this can also include nontangible, transitorycomputer-readable media, such as data signals, data transmissions, orany other medium which can be used to transmit the desired informationand which can be accessed by computing system 1900.

By way of example, computer-readable storage media 1922 may include ahard disk drive that reads from or writes to non-removable, nonvolatilemagnetic media, a magnetic disk drive that reads from or writes to aremovable, nonvolatile magnetic disk, and an optical disk drive thatreads from or writes to a removable, nonvolatile optical disk such as aCD ROM, DVD, and Blu-Ray® disk, or other optical media.Computer-readable storage media 1922 may include, but is not limited to,Zip® drives, flash memory cards, universal serial bus (USB) flashdrives, secure digital (SD) cards, DVD disks, digital video tape, andthe like. Computer-readable storage media 1922 may also include,solid-state drives (SSD) based on non-volatile memory such asflash-memory based SSDs, enterprise flash drives, solid state ROM, andthe like, SSDs based on volatile memory such as solid state RAM, dynamicRAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, andhybrid SSDs that use a combination of DRAM and flash memory based SSDs.The disk drives and their associated computer-readable media may providenon-volatile storage of computer-readable instructions, data structures,program modules, and other data for computer system 1900.

Communications subsystem 1924 provides an interface to other computersystems and networks. Communications subsystem 1924 serves as aninterface for receiving data from and transmitting data to other systemsfrom computer system 1900. For example, communications subsystem 1924may enable computer system 1900 to connect to one or more devices viathe Internet. In some aspects, communications subsystem 1924 can includeradio frequency (RF) transceiver components for accessing wireless voiceand/or data networks (e.g., using cellular telephone technology,advanced data network technology, such as 3G, 4G or EDGE (enhanced datarates for global evolution), WiFi (IEEE 802.28 family standards, orother mobile communication technologies, or any combination thereof),global positioning system (GPS) receiver components, and/or othercomponents. In some aspects, communications subsystem 1924 can providewired network connectivity (e.g., Ethernet) in addition to or instead ofa wireless interface.

In some aspects, communications subsystem 1924 may also receive inputcommunication in the form of structured and/or unstructured data feeds1926, event streams 1928, event updates 1919, and the like on behalf ofone or more users who may use computer system 1900.

By way of example, communications subsystem 1924 may be configured toreceive unstructured data feeds 1926 in real-time from users of socialmedia networks and/or other communication services such as Twitter®feeds, Facebook® updates, web feeds such as Rich Site Summary (RSS)feeds, and/or real-time updates from one or more third party informationsources.

Additionally, communications subsystem 1924 may also be configured toreceive data in the form of continuous data streams, which may includeevent streams 1928 of real-time events and/or event updates 1919, thatmay be continuous or unbounded in nature with no explicit end. Examplesof applications that generate continuous data may include, for example,sensor data applications, financial tickers, network performancemeasuring tools (e.g. network monitoring and traffic managementapplications), clickstream analysis tools, automobile trafficmonitoring, and the like.

Communications subsystem 1924 may also be configured to output thestructured and/or unstructured data feeds 1926, event streams 1928,event updates 1919, and the like to one or more databases that may be incommunication with one or more streaming data source computers coupledto computer system 1900.

Computer system 1900 can be one of various types, including a handheldportable device (e.g., an iPhone® cellular phone, an iPad® computingtablet, a PDA), a wearable device (e.g., a Google Glass® head mounteddisplay), a PC, a workstation, a mainframe, a kiosk, a server rack, orany other data processing system.

Due to the ever-changing nature of computers and networks, thedescription of computer system 1900 depicted in the figure is intendedonly as a specific example. Many other configurations having more orfewer components than the system depicted in the figure are possible.For example, customized hardware might also be used and/or particularelements might be implemented in hardware, firmware, software (includingapplets), or a combination. Further, connection to other computingdevices, such as network input/output devices, may be employed. Based onthe disclosure and teachings provided herein, a person of ordinary skillin the art will appreciate other ways and/or methods to implement thevarious aspects.

In the foregoing specification, aspects of the invention are describedwith reference to specific aspects thereof, but those skilled in the artwill recognize that the invention is not limited thereto. Variousfeatures and aspects of the above-described invention may be usedindividually or jointly. Further, aspects can be utilized in any numberof environments and applications beyond those described herein withoutdeparting from the broader spirit and scope of the specification. Thespecification and drawings are, accordingly, to be regarded asillustrative rather than restrictive.

What is claimed is:
 1. A system comprising: a non-transitorycomputer-readable medium storing computer-executable programinstructions; and a processing device communicatively coupled to thenon-transitory computer-readable medium for executing thecomputer-executable program instructions, wherein executing thecomputer-executable program instructions configures the processingdevice to perform operations comprising: creating a discourse tree fromfragments of text, wherein the discourse tree comprises a plurality ofnodes, each non-terminal node representing a rhetorical relationshipbetween two of the fragments of text, and each terminal node isassociated with one or more fragments and is associated with anon-terminal node; for each non-terminal node of the discourse tree:identifying, from the discourse tree, a rhetorical relationshipassociated with the non-terminal node, accessing a rule corresponding tothe identified rhetorical relationship, wherein the rule identifies forselection, based on the rhetorical relationship, one or more of (i) acorresponding nucleus elementary discourse unit or (ii) a correspondingsatellite elementary discourse unit, and selecting, based on the rule,one or more of (i) the fragment of text associated with the nucleuselementary discourse unit or the (ii) fragment of text associated withthe satellite elementary discourse unit; and creating a searchable indexcomprising multiple entries, each entry corresponding to a selectedfragment.
 2. The system of claim 1, wherein executing thecomputer-executable program instructions configures the processingdevice to perform operations comprising: determining, for each fragmentof text, a question; receiving a query from an external device;searching, in each of the questions for matches corresponding to thequery; and responsive to determining a match in the questions, providinga response to the external device.
 3. The system of claim 1, whereinexecuting the computer-executable program instructions configures theprocessing device to perform operations comprising, for at least onenon-terminal node of the discourse tree: identifying the rhetoricalrelationship as either (i) a contrast or (ii) an elaborationrelationship; and responsive to determining that the rule specifiesaccessing the satellite elementary discourse unit, selecting thesatellite elementary discourse unit.
 4. The system of claim 1, whereinthe rule specifies (i) selecting the fragments associated with thenucleus elementary discourse unit and (ii) discarding fragmentsassociated with the satellite elementary discourse unit.
 5. The systemof claim 1, wherein executing the computer-executable programinstructions configures the processing device to perform operationscomprising, for at least one non-terminal node of the discourse tree:identifying the rhetorical relationship as either (i) enablement, (ii)elaboration, or (iii) contrast; and responsive to determining that therule specifies accessing the nucleus elementary discourse unit,selecting the nucleus elementary discourse unit.
 6. The system of claim1, wherein each rhetorical relationship is one of condition, contrast,or attribution.
 7. The system of claim 1, wherein executing thecomputer-executable program instructions configures the processingdevice to perform operations comprising: determining, for each fragment,a question; and providing the questions to an external device.
 8. Thesystem of claim 1, wherein executing the computer-executable programinstructions configures the processing device to perform operationscomprising, for at least one non-terminal node of the discourse tree:identifying the rhetorical relationship as either (i) a same unit or a(ii) joint relationship; and responsive to determining that the rulespecifies accessing both (i) the nucleus elementary discourse unit and(ii) the satellite elementary discourse unit, selecting both (i) thenucleus elementary discourse unit and (ii) the satellite elementarydiscourse unit.
 9. The system of claim 1, wherein executing thecomputer-executable program instructions configures the processingdevice to perform operations comprising, for at least one non-terminalnode: identifying the rhetorical relationship as an attributionrelationship; and responsive to determining that the rhetoricalrelationship does not represent a query by an author, selecting thenucleus elementary discourse unit.
 10. A system comprising: anon-transitory computer-readable medium storing computer-executableprogram instructions; and a processing device communicatively coupled tothe non-transitory computer-readable medium for executing thecomputer-executable program instructions, wherein executing thecomputer-executable program instructions configures the processingdevice to perform operations comprising: creating, for an answer of aquestion-answer index comprising fragments of text, a discourse tree,wherein the discourse tree comprises a plurality of nodes, eachnon-terminal node representing a rhetorical relationship between two ofthe fragments of text and each terminal node is associated with one ormore fragments of text; for each non-terminal node in the discoursetree: identifying, from the discourse tree, a rhetorical relationshipassociated with the non-terminal node, accessing a rule corresponding tothe identified rhetorical relationship, wherein the rule identifies forselection, based on the identified rhetorical relationship, either acorresponding nucleus elementary discourse unit or a correspondingsatellite elementary discourse unit, and selecting, based on the rule,one or more of (i) the fragment of text associated with the nucleuselementary discourse unit or (ii) the fragment of text associated withthe satellite elementary discourse unit; and creating an index ofadditional questions comprising an entry corresponding to the selectedfragment of text.
 11. The system of claim 10, wherein executing thecomputer-executable program instructions configures the processingdevice to perform operations comprising: receiving a query from anexternal device; searching, in each question of the question-answerindex, for a match corresponding to the query; responsive to determiningthat fewer than a threshold number of matches are available, searchingfor additional matches to the query in the index of additionalquestions; responsive to determining that fewer than an additionalthreshold number of additional matches are available in the index ofadditional questions, searching for further matches in each answer inthe question-answer index; and providing the matches to the externaldevice.
 12. The system of claim 10, wherein executing thecomputer-executable program instructions configures the processingdevice to perform operations comprising, for at least one non-terminalnode: identifying the rhetorical relationship as either (i) a same unitor a (ii) joint relationship, and selecting both (i) the nucleuselementary discourse unit and (ii) the satellite elementary discourseunit, wherein the rule specifies accessing both (i) the nucleuselementary discourse unit and (ii) the satellite elementary discourseunit.
 13. The system of claim 10, wherein executing thecomputer-executable program instructions configures the processingdevice to perform operations comprising, for at least one non-terminalnode of the discourse tree: identifying the rhetorical relationship asan attribution relationship, and responsive to determining that therhetorical relationship does not represent a query by an author,selecting the nucleus elementary discourse unit.
 14. The system of claim10, wherein each entry represents a question that can be answered by theanswer.
 15. The system of claim 10, wherein the rule specifies (i)selecting fragments of text associated with the nucleus elementarydiscourse unit and (ii) discarding fragments of text associated with thesatellite elementary discourse unit.
 16. The system of claim 10, whereinexecuting the computer-executable program instructions configures theprocessing device to perform operations comprising: receiving a queryfrom an external device; searching, in the index of additional questionsfor matches corresponding to the query; and responsive to determining amatch in the index of additional questions, providing a response to theexternal device.
 17. The system of claim 10, wherein executing thecomputer-executable program instructions configures the processingdevice to perform operations comprising, for at least one non-terminalnode: identifying the rhetorical relationship as either (i) a contrastor (ii) an elaboration relationship; and responsive to determining thatthe rule specifies accessing the satellite elementary discourse unit,selecting the satellite elementary discourse unit.
 18. A non-transitorycomputer-readable storage medium storing computer-executable programinstructions, wherein when executed by a processing device, thecomputer-executable program instructions cause the processing device toperform operations comprising: training a classification model byiteratively: providing one of a set of training data pairs to theclassification model, wherein each training data pair comprises text andan expected classification of one or more of: a nucleus elementarydiscourse unit of a training discourse tree or a satellite elementarydiscourse unit of the training discourse tree; receiving, from theclassification model, a determined classification that is indicative ofeither a nucleus elementary discourse unit or a satellite elementarydiscourse unit; calculating a loss function by calculating a differencebetween the determined classification and the expected classification;and adjusting internal parameters of the classification model tominimize the loss function; creating a discourse tree from fragments ofan additional text, wherein the discourse tree comprises a plurality ofnodes, each non-terminal node representing a rhetorical relationshipbetween two of the fragments of the additional text and each terminalnode is associated with one or more of the fragments of the additionaltext; and for each terminal node in the discourse tree, labeling theterminal node as either a nucleus elementary discourse unit or asatellite elementary discourse unit by applying the trainedclassification model to fragments of the additional text associated withone or more of the terminal nodes.
 19. The non-transitorycomputer-readable storage medium of claim 18, wherein when executed by aprocessing device, the computer-executable program instructions causethe processing device to perform operations comprising: creating anadditional discourse tree from a body of text comprising fragments,wherein the additional discourse tree comprises a plurality of nodes,each non-terminal node representing a rhetorical relationship betweentwo of the fragments and each terminal node associated with one or morefragments; for each non-terminal node in the additional discourse tree:identifying a rhetorical relationship associated with the non-terminalnode; providing a fragment of text associated with the non-terminal nodeand the rhetorical relation to the classification model; and receiving,from the classification model, a classification indicating whether thefragment of text is informative or not informative; and creating asearchable index comprising each of the fragments identified asinformative.
 20. The non-transitory computer-readable storage medium ofclaim 19, wherein when executed by a processing device, thecomputer-executable program instructions cause the processing device toperform operations comprising: receiving a query from an externaldevice; searching, in each of question of a plurality of question-answerpairs, for matches corresponding to the query; responsive to determiningthat fewer than a threshold number of matches are available, searching,in the searchable index, for additional matches to the query; responsiveto determining that fewer than a threshold number of additional matchesare available in the searchable index, searching for further matches ineach answer of the plurality of question-answer pairs; and providing aresponse to the external device.